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ABSTRACT 

A study evaluated the effectiveness of a 
teacher-researcher program begun in the Fort Worth Indpendent School 
District to improve and assess students* writing abilities. The study 
sought to determine if there was a noticeable improvement in the 
writing skills of students studying with the teacher-researchers as 
opposed to students studying with nonpart icipants . All teachers had 
their students complete two pieces of timed writing on topics 
previously proved successful in the district. The holistic method, 
through which readers are encouraged to view a piece of writing as 
more than the mere sum of its parts, was used to assess the papers. 
Considered as a group, writing samples submitted by students of 
teacher-researchers showed an improvement in average scores. In five 
of the six grades evaluated, the writing of teacher-researchers* 
students showed greater improvement than did the writing of the 
comparison group, but the writing of the comparison group students 
showed improvement in only two of the six grade levels. (Tables of 
data are included.) (NKA) 
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Beginning a Teacher-Researcher Program: The First Steps 

With growing concern over the writing abilities of students from 
first through twelfth grades, school districts are constantly seeking 
models by which to improve and assess the writing abilities of their 
students. In Texas' Fort Worth Independent School District, the desire 
to increase literacy— and to implement the roost reliable and valid means 
by which to establish increased 1 iteracy—prompt^jd the administration to 
approve a Teacher-Researcher project in the sunmer of 1985 that would, if 
successful, serve as a model for further in-service teacher training in 
writing. In the following few pages, we would like briefly to summarize 
our project and its evaluation methods in order to introduce other teachers 
and administrators to a system that premises to yield initial information 
about the effectiveness of writing programs. 

The Teacher-Researcher movement, as chronicled and defined by Dixie 
Goswami, tells us that a research-based approach to teaching writing allows 
teachers to become theorists who articulate and test their instructional 
assumptions. As they form coimunUies that share the efforts and outcomes 
of research, Teacher-Researchers are able to generate information critical 
to our answering pressing questions about the complex, subtle realities of 
the classroom. Structured ypon the beliefs of the Teacher-Researcher 
movement, our program consisted of three stages. First, our program 
participants--twenty-one Fort Worth I.S.D. kindergarten through twelfth- 
grade teachers— received three weeks of intensive training in the theory 
and practice of writing instruction, ways by which to help students 
understand and implement processes conducive to successful writing. We 
also examined current methods of documenting and evaluating data concerring 
writing and learning processes. Second, these teachers returned to the 
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classroom the next fall to apply the techniques in which they had been 
trained and to gather data on specific questions about the writing-learning 
process that they had formulated and wished to answer through research. 
Third, the teachers reassembled periodically to share their data gathering 
procedures and research findings with what was to become a community of 
Teacher-Researchers. 

In an effort to discern the eff ectivensss of our training program and 
to train our teachers further in evaluation methods, we gathered assessment 
data on our own program. The purpose of our investigation was to discover 
if there was a noticeable improvement in the writing skills of students 
studying with our Teacher-Researchers as opposed to students studying with 
non-participants. We matched the Teacher-Researchers' students with the 
students of teachers who had not attended the intensive training workshops 
but who volunteered to participate in out study. In all cases, the 
Teacher-Researchers were matched with teachers of comparable experience 
and their students were matched with students of comparable grade levels 
and socio-economic status. 

We asked all teachers concerned to have their students complete two 
pieces of timed writing on topics previously proven successful in the 
District. The first of these timed writings occurred in October of 1985 
and served as cur pre-test; the second occurred in May of 1986 and 
served as our post-test. Fro-s the 2200 writing samples submitted after 
each timed-writing, 468 papers, a random sampling, were pulled for 
assessm'jnt: 288 papers from the students of Teacher-Researchers and 180 
papers from the students of non-participants. (The difference in the 
number of papers pulled from our two groups simply reflects the fact 
that a greater percentage of the 2200 contributing students were 
students of Teacher-Researchers.) 
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We adopted the holistic method of assessing these papers, a method 
through which readers are encouraged to view a piece of writing as more 
than the mere sum of its elementary parts. (For more on the issues 
theories, and practice of holistic evaluation, see: Cooper and Odell; 
Myers; Ruth and Murphy; White; Greenberg, Wiener, and Donovan.) 
In order to prepare themselves for the assessment, one week before the 
evaluation the Teacher-Researchers spent one day making test development 
decisions. The teachers reviewed papers from each of the grade levels 
assessed— first, third, sixth, eighth, ninth, eleventh, and twelfth— 
in order to select sample training papers for the reader training session. 
These training papers were chosen to represent the compositional 
effectiveness of each level of the six-point assessment scale that we 
used, as well as to suggest the variety of papers that could occur at each 
level. In the process of selecting these papers, our Teacher-Researchers 
also described the compositional traits of each level of the six-point 
scale— trails such as topic development, unity, sentenct variety, word 
choice, punctuation, and grammar. The articulation of these traits 
developed into a scoring guide that would supplement the sample training 
papers in the readers' training session to take place just before scoring 
bsgan. 

It is the practice in holistic scoring to have each paper read and 
scored by two readers and to award the paper a final score that is the sum 
of its two scores. Therefore, in addition to selecting the training 
papers and developing the scoring guide^our Teacher-Researchers 
determined what degree of discrepancy between two scores was acceptable. 
Our participants agreed that papers receiving scores that differed by 
more than two points (i.e.,: 6+3, 5+2, and so on) would be given to a 
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third reader and the paper would be awarded the sum of the third reader's 
score and the previous score closer to that. Additionally, based on the 
quality of the writing occurring at each level, our Teacher-Researchers 
decided that the minimum acceptable level of writing for this assessment, 
the cut-score, would be a cumulative score of seven. 

When the scoring day arrived, our readers— District teachers whose 
students were not involved in the assessment— were trained with the 
Teacher-Researchers' sample training papers and scoring guide in the hour 
before actual scoring began. At the end of the morning's work in which 
468 papers were scored, we found that we had achieved outstanding 
inter-reader reliability for our assessment: 

Table 1 

Percentage of Agreement Among Evaluators 

Pre-Tests Post-Tests 



Grade 1 


97% 


100% 


Grade 3 


95% 


92% 


Grade 6 


94% 


97% 


Grade 8 


92% 


92% 


Grade 9 


91% 


98% 


Grade 11 


82% 


100% 


Grade 12 


100% 


100% 



Since 80% inter-reader reliability is considered acceptable in an 
holistic evaluation, we were pleased that our scoring was so very accurate, 
and we then felt secure in further analysis of the data in order to see 
if and how the average scores and the cut scores of the two groups of 
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evaluated students differed. 

Considered as a group, writing samples submitted by students of 
Teacher-Researchers showed an improvement in average scores, as the table 
belcw indicates: 

Table 2 

Comparison of Averages of Students' Scores 

Teacher-Researchers' Comparison Group Students 

Students 





Average Score 
on Pre-Test 


Average Score 
on Post-Test 


Average Score 
on Pre-Test 


Average Score 
on Post-Test 


Grade 1 


8.2 


9.2 


6.8 


7.4 


Grade 3 


4.7 


5.4 


6.6 


6.4 


Grade 6 


6.0 


8.3 


7.7 


6.8 


Grade 8 


6.0 


6.6 


5.4 


5.8 


Grade 9 


6.6 


7.5 


4.1 


3.9 


Grade 11 


8.6 


8.3 


8.7 


8.3 



Although writing samples from students in the control group revealed the 
average post- test score that was .12 of a point lower than the average 
pre-test score, the average score the the post-tests of Teacher-Researchers' 
students was .87 of a point higher than the average score for their pre- 
tests. Indeed, this pattern of improvement of the students of Teacher- 
Researchers is consistent across grade levels. In five of the six grades 
evaluated, the writing of Teacher-Researchers' students showed greater 
improvement than did the writing of the compar-ison group, but the writing 
of the comparison group students showed improvement in only two of the six 
grade levels. 
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(As we discussed above, we were finable to match the number of 

students from each level, but we were unable to find any comparison 

group whatsoever for cur Teacher-Researchers' sixth grade special education 

and twelfth grade students. We considered it important, however, to 

assess both of these groups so that we would have a more complete picture of the 

effects of our program. Consistent with our other populations, we found 

improvement in both levels: the average pre- test score for the sixth graders 

was 5.6, while the average post-test score was 8.25. Similarly, the average 

pre-test score for the twelfth graders was 5.8, while the post-test scores 

were 7.0. ) 

As a group, the writing samples from students of our Teacher-Researchers 
also showed considerable improvement in the increased number of p "ers whose 
scores totaled seven: 

Table 3 

Comparison of % of Essays Receiving a Score of 7 or Higher 



Teacher Researchers' Students Comparison Group Students 
Pre-Test Post Test Pre-Test Post-Test 





% of Papers 
with 7 or 
higher 


% of Papers 
with 7 or 
higher 


% of Papers 
with 7 or 
higher 


% of Papers 
with 7 or 
higher 


Grade 1 


71% 


100% 


61% 


6'% 


Grade 3 


17% 


35% 


43% 


43% 


Grade 6 


37.5% 


80% 


68% 


55% 


Grade 8 


45% 


42.1% 


33.1% 


38.9% 


Grade 9 


50% 


69% 


11% 


11% 


Grade 11 


81% 


77% 


85% 


82% 



On the pre-test samples, an average of 50.25% of the Teacher-Researchers' 
students received a score of 7 or better, and on the post-test an average 
of 67.18% of these students received a score of at least /. By contrast. 
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scores of comparison group students showed an overall decrease. On their 
pre- tests, the average percentage of papers with acceptable scores was 
50.18«— a percentage suggesting that these students were of comparable 
writing ability with the Teacher-Researchers' students— yet the post-test 
writing samples of students in the control group revealed that their scores 
actually had decreased to 49.48%. Moreover, just as the average scores 
of the Teacher-Researchers' students had increased across grade levels, 
so too did these students illustrate gains in their ability to achieve the 
cut score of seven. That is, among the students of Teacher-Researchers, 
four of the six grade levels showed an increase in the percentage of 
students who achieved a minimally acceptable score, while only two of the 
six grade levels of the comparison groups revealed an increase. 

(As in the case of their average scores, the level of acceptability 
of the sixth grade special education students and the twelfth grade 
students showed substantial increase. While the sixth grade students 
showed in pre-test that only 33% had achieved a score of 7 or above, 83% of 
the students had achieved this score or better on the Dost-test. Similarly, 
only 41% of the twelfth orade students had achieved the cut-score in the 
pre-test, but upon post-test 66% were able tc achieve at least a score of 
7.) 

We found, then, that the writing of our Teacher-Researchers' ?tudents 
showed substantial gains in both average scores and increased level of 
competency. We wore also aware, though, that this form of evaluation 
would not, taken by itself, indicate the processes of instruction that the 
students received. Are teachers, for example, having their students write 
in different modes for different audiences? Are the students practicing 
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revision? P^er evaluation? Clearly, these concerns also had to be addressed. 

In order to provide a record of Instruction regarding how the writing 
process was taught, we asked our Teachar-Researchers to keep a writing 
folder over the year for each student. Included in each folder were to be 
the results of at least six process-oriented writing assignments: a 
copy of the assignment, notes (or other informational writing) by which the 
students gathered information, an initial draft, a reader commentary sheet 
completed either b> peer or teacher, and the revised draft. In order to 
assess the folders themselves, we simply designed a content-specific 
checklist which would allow us to discover information—peer revision 
sheets, for Instance, or the absence of them--about how the process approach 
of teachii g writing was being executed. Although we are still examining and 
refining the folder evaluation part of our Project, we have found that 
teachers can practically maintain writing folders for their students and 
that these fc'ders will yield more complete information about the students' 
writing ability than will, say, the results of a multiple choice end of 
semester test. 

Thus we had employed two measures—one product, the other process 
oriented—in order to better assess the writing abilities of those students 
studying with our Teacher-Researchers. We found improvement in these 
students, and we believe that we discovered facets of program development 
just as significant for the teachers themselves and their District. We 
will close with three of our findings. 

The best program development practices incorporate patience. 

In implementing the Teacher-Researcher Project, we came to realize 
that it would take time to gather definitive information on our methods and 
their results, and thus we came to realize that "problems" with our initial 
assessment attempt are only understood to be "weaknesses" if we hold that 
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we havG "irrefutable proof" that our methods produce stronger writing 
abilities in students. At present, we would only claim that the results 
of this year's work are encouraging and that, over a period of time, we 
will discover more exact methods by which to te:ch, research, and evaluate 
student writing. 

Patience affords a sense of community . 

Our Teacher-Researchers, in pursuing both their individual research 
projects and our holistic evaluation, have become a more unified group« 
They conmunicate with each other frequently to discuss how to combine ideas 
and methods, and so the narmful boundaries which separate theory from 
practice begin to dissolve: our Teacher-Researchers formulate imaginative 
heuristics, test their assumptions, and refine them to create other 
imaginative heuristics. The^e teachers, therefore, believe themselves to 
be neither mechanical pedagogues nor abstract theorists; simply and 
wonderfully, they see themselves as producers of knowledge. 
A sense of community restores confidence . 

Mandatory comoetency testing for both teachers and their students, 
criteria-based classroom procedures, too much paper worK and too little 
money— these are issues in the lives of all teachers which often seem more 
likely to bury education than improve it. Yet the sense of cofidence 
through community implicit in the Teacher-Researcher concept encourages 
the classroom teacher to remember that teaching is a vocation and not 
merely a job. We have found that our Teacher-Researchers strive to find 
creative, intelligent ways to integrate educational issues into their 
classrooms. For instance, instead of abandoning the assessment of their 
students to so-called specialists, our Teacher-Researchers have found that 
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they are able to reliably and Validly assess large samples of writing 
through direct methods. Instead of viewing assessment as a singular, 
closed act, our Teacher- Researchers are ab'ie to refine the by-products 
of their research- strategies for questi6n design, methods for selecting 
sample papers, ways for describing the elements of these papers, ideas 
for using these models and descriptors back in the classroom— and to 
share these by-products with other teachers. In other words, our Teacher- 
Researchers realize the value of their work and are confident enough in 
its pursuit to involve other educators in classroom-based inquiry. 

As we begin the second year of our Teacher-Researcher Project, we are 
encouraged most in that our Teacher-Researchers have begun to establish 
the mean: by which school districts are turned into investigative coninunities. 
We believe that our Project can help to create an environment in which 
occur the everyday miracles of transforniation. 
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